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(57) A three-dimensional space is generated using 
shape model data and its surface characteristic and pat- 
tern, a space is rebuilt from the three-dimensional space 
as an image in accordance with a drawing specification 
determined by the viewpoint position, direction, angle of 



view, and the like, an image is generated from light 
space data using the drawing specification, and the im- 
ages generated based on the shape model data and the 
light space data are fused, thereby easily generating 
and fus.ng shape model data and an actually taken im- 
age, and displaying an image with high reality. 
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Description 

The present invention relates to an image process- 
ing method and apparatus for generating and displaying 
a virtual environment (or virtual reality. 5 

As a conventional method of expressing a three-di- 
mensional object and space, and presenting a view im- 
age from an arbitrary position and direction, the follow- 
ing methods are known: 

to 

(1) A three-dimensional object or space is ex- 
pressed using shape model data such as polygon 
data, curved surface data, and the like, texture data 
representing the surface attribute or pattern, light 
source data, and the like, and the view image of the '5 
space from an arbitrary position and direction is 
drawn by the rendering technique of computer 
graphics. 

(2) Upon creating a three-dimensional virtual envi- 
ronment using the conventional method (1), ele- 20 
ments (coordinate transformation data, shape data, 
surface attribute data, illumination, and the like) that 
make up the virtual environment are expressed by 

a tree structure. That is, a space, ground, architec- 
ture, room, furniture, illumination, ornament, and 2s 
the like that make up the three-dimensional space 
originally have a hierarchical nesting relationship 
thereamong. For example, an ornament on a table 
depends on the table arrangement so that it moves 
together with the table arrangement, and it is often 30 
convenient to arrange such ornament relative to the 
coordinate system of the table. For this reason, a 
data structure having hierarchical dependence on 
the arrangement is used. As a method of express- 
ing such structure, a virtual environment is ex- 3S 
pressed by an n-ary tree structure. 

For example, Fig. 18 shows an illustration ex- 
ample of a certain simple virtual environment. In the 
case of this figure, paying attention to a room, table, 
and sofa, the room is described on a coordinate sys- 40 
tern C 2 transformed from a world coordinate system 
C 0 by a coordinate transformation T 2 , and the table 
and sofa in the room are respectively described on 
coordinate systems C 3 and C 4 transformed from the 
coordinate^ system C 2 by coordinate translorma- 45 
lions T 3 and T 4 . A pot on the table is described on 
a coordinate system C 5 transformed from the coor- 
dinate system C 3 by a coordinate transformation T 5 . 
Furthermore, ray (or light) space data is arranged 
on the desk. This data is described on a coordinate so 
system C 6 transformed from the coordinate system 
C 3 by a coordinate transformation T6 as in the pot. 
When these objects arc expressed by a typical tree 
structure, a tree shown in Fig. 19 is obtained. 
(3) The images of a three-dimensional object or ss 
space are taken in advance from a large number of 
viewpoints, an image taken under a phototaking 
condition close to a desired view position and direc- 



tion is selected from the taken images, and a three- 
dimensional object viewed from the position and di- 
rection close to the view position and direction is 
displayed, thereby approximately expressing a 
view from an arbitrary position and direction 

(4) Ray space data is generated on the basis of the 
actually taken images of a three-dimensional obiect 
or space, and an image viewed from an arbitrary 
position and direction is generated and displayed 
on the basis of the ray space data, thereby recon- 
structing the three-dimensional object or space. 

In this method, an object is expressed as a 
set of light components emanating from the object 
without calculating the shape of the object 

(5) A panorama image obtained by looking around 
from a given viewpoint is input, and an image cor- 
responding to the view direction of the viewer is 
generated based on the panorama image (mainly 
attained by extracting a partial image from the pan- 
orama image and correcting distortion of the ex- 
tracted image), thereby displaying a three-dimen- 
sional space from a given point. 

However, the above-mentioned conventional meth- 
ods (1 ) to (5) suffer the following problems. 

It is difficult for the conventional method (1 ) to gen- 
erate or reconstruct the shape data of an object having 
a very complicated shape. Furthermore, it is also difficult 
for the method (1 ) to acquire the shape data of an object 
with a complicated shape from a real object using a 
three-dimensional measurement apparatus. In particu- 
lar, it is more difficult for the method ( 1 ) to reconstruct a 
real object having an existing complicated shape or a 
complicated surface pattern or reflection characteristics 
(absorption/transmission characteristics). Furthermore, 
the method (1 ) is generally easy to express an artificial 
object but is hard to express a natural object. However, 
this method has a merit, i.e., it can express an artificial, 
simple three-dimensional space such as a room, a row 
of stores and houses, or the like, which is mainly built of 
planes with a small data volume. 

The conventional method (2) is an expression/de- 
scription method of data, and suffers the problems of 
the conventional method ( 1 ) However, this expression/ 
description method of data is an excellent one. 

In the conventional method (3), the above-men- 
tioned problems are not posed. However, since the im- 
ages to be finally presented must be taken in advance, 
a very large number of images must be prepared and a 
huge data volume is required, so as to artificially obtain 
an arbitrary viewpoint position and direction. In view of 
the data volume and phototaking required for obtaining 
a large number of images, it is impossible to put this 
method into practical applications. For the same reason, 
it is nearly impossible to hold every data to express a 
wide three-dimensional space such as a room, a row of 
stores and houses, and the like This method is suitable 
for expressing a lhree<iimensional object by taking the 
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images of the object from its surrounding positions 

In the conventional method (4). a large number of 
images need not be taken in advance unlike in the con- 
ventual method (3). Once ray space data is generated 
based on a predetermined number of images taken in 
advance, a view .mage from an arbitrary viewpoint po- 
sition can be generated (strictly speaking, there is a con- 
straint condition). However, in order to present images 
from every positions in a three-dimensional space a 
huge volume of ray space data must also be generated 
and held. This method is also suitable for a ihree-dimen- 
sional object as in the conventional method (3) but is 
not suitable for expressing a three-dimensional' space 
such as a room, a row of stores and houses, or the like 
The conventional method (5) is suitable for express 
mg a three-dimensional space such as a room a row of 
stores and houses, or the like, and a view image in an 
arbitrary direction can be presented as long as the view- 
point position is limited. However, when the viewpoint 
position ,s to be arbitrarily moved, panorama images 
from a very large number of viewpoint positions must be 
prepared in advance to express an arbitrary movement 
by approximation. Accordingly, a very large number of 
panorama images must be prepared. For this reason it 
•s difficult in practice to attain viewing from an arbitrary 
viewpoint position owing to a huge data volume and dif- 
ficulty ,n phototaking. Hence, this processing is normally 
realized by reducing the data volume and allowing only 
discrete movements of the viewpoint position 

Furthermore, the conventional methods (1 ) and (2) 
and the methods (3). (4). and (5) are fundamentally dif- 
ferent techniques, and there is no method that can utilize 
their characteristics and combine these methods by ef- 
fectively using only their merits. 

An aspect of the present invention provides an im- 
age processing method and apparatus, which can utilize 
the characteristics of the conventional methods (1) and 
(2). and methods (3), (4). and (5), which are originally 
different techniques, and can combine these methods 
to effectively take an advantage of only their merits. 

An image processing method according to one as- 
pect of the present invention is an image processing 
method of generating and displaying a virtual environ- 
ment, comprising: 



crated in the actually taken image-based imaoe 
generation step 

An image processing apparatus according (oanoth 
5 or aspect of the present invention ,s an imaqc process- 
ing apparatus for generating and displaying a virtual en- 
vironment, comprising: 



10 



is 



20 



25 



30 



35 



40 



model space rebuild.ng means lor generating a 
three-dimensional space using shape model data 
and rebuilding the space as an image from the 
three-dimensional space in accordance with a first 
drawing specification at a given viewpoint 
actually taken image-based image generation 
means for generating an image from an arbitrary 
viewpoint on the basis of an actually taken .mage in 
accordance w.th a second drawing specification 
that uses the same view condition as a view condi- 
tion used by the first drawing specification: and 
image fusion means for (using the image rebuilt by 
the model space rebuilding means and the image 
generated by the actually taken image-based im- 
age generation means. 

An imago processing method according to another 
aspect of the present invention is an image processing 
method of generating a virtual environment, comprising: 

the first description step of describing a shape of a 
three-dimensional object and a shape of a three- 
dimensional space on the basis of information in- 
cluding a three-dimensional position, a three-di- 
mensional direction, and a surface color: 
the second description step of describing a three- 
dimensional object and a three-dimensional space 
on the basis of an actually taken image taken from 
an arbitrary position; and 

the generation step of generating the virtual envi- 
ronment by fusing the shapes described in the first 
description step and the three-dimensional object 
and the three-dimensional space described in the 
second description step. 



the model space rebuilding step of generating a 
three-dimensional space using shape model data 
and rebuilding the space as an image from the 
three-dimensional space in accordance with a first 
drawing specification at a given viewpoint; 
the actually taken image-based image generation 
step of generating an image from an arbitrary view- 
point on the basis of an actually taken image in ac- 
cordance with a second drawing specification that 
uses the same view condition as a view condition 
used by the first drawing specification; and 
the image fusion step of fusing the image rebuilt m 
the model space rebuilding step and the image gen- 



An image processing apparatus according to a fur- 
■« theraspeci of the present invention is an image process- 
ing apparatus lor generating a virtual environment, com- 
prising: 



so 



55 



first description means for describing a shape of a 
three-dimensional object and a shape of a three- 
dimensional space on the basis of information in- 
cluding a three-dimensional position, a three-di- 
mensional direction, and a surface color, 
second description means for describing a three- 
dimensional object and a three-dimensional space 
on the basis of an actually taken image taken from 
an arbitrary position; and 

generation means for generating the virtual envi- 
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ronmenl by fusing the shapes described by the first 
description means and the three-dimensional ob- 
ject and the three-dimensional space described by 
the second description means. 

5 

Embodiments of the present invention will now be 
described with rclerence to the accompanying draw- 
ings, in which: 

Fig. 1 is a block diagram of a virtual environment w 
generation realization apparatus according to the 
first embodiment ol the present invention; 
Fig. 2 which is composed of Figs. 2A and 2B is a 
flow chart showing the flow of the processing of the 
first embodiment; is 
Figs. 3A and 3B are views of a three-dimensional 
space lor explaining the principle of a light space; 
Figs, 4A and 4B are views showing the relationship 
between light rays in a real space and light space; 
Figs. 5A and 5B are views showing the principle of 20 
inter-transformation between the real space and 
light space; 

Fig. 6 is a schematic view showing the dependence 
among coordinate systems that express arrange- 
ment data of light space data; 2s 
Fig. 7 is a schematic view showing the region for 
performing drawing discrimination of one arrange- 
ment data of light space data; 
Fig. 8 is a view showing an example of the screen 
image actually generated according to the first em- 30 
bodiment; 

Fig. 9 is a view showing an example of the screen 
image actually generated according to the first em- 
bodiment; 

Fig. 10 is a view showing an example of the screen os 
image actually generated according to the first em- 
bodiment; 

Fig. 11 is a view showing an example of the screen 
image actually generated according to the first em- 
bodiment; 40 
Fig. 1 2 is a view showing an example of the screen 
image actually generated according to the first em- 
bodiment; 

Fig. 1 3 is a v^w showing the principle of generating 
an image based on multi-viewpoint images used in -*s 
the second embodiment; 

Fig. 14 is a flow chart showing image generation 
based on multi-viewpoint images used in the sec- 
ond embodiment; 

Fig. 15 is a view showing the principle ol distortion so 
correction in the vertical direction in the image gen- 
eration of the second embodiment; 
Fig. 1 6 is a block diagram showing the third embod- 
iment of the present invention; 

Figs. 17A and 17B are flow charts of the third em- ss 
bodiment; 

Fig 18 is a view showing an illustration example of 
a virtual environment built by the conventional 



method (2); 

Fig 19 is a view that expresses the virtual environ- 
ment shown in Fig. 18 by a tree structure; 
Fig. 20 is a view showing an illustration example of 
a virtual environment generated by the third embod- 
iment; 

Fig. 21 is a view that expresses the virtual environ- 
ment shown in Fig. 20 by a tree structure; 
Fig. 22 is a view showing data that express the tree 
structure shown in Fig. 20 by text data; and 
Fig. 23 is a view showing the virtual environment 
being viewed in the third embodiment. 

(a) First Embodiment: 

Fig. 1 is a block diagram of a virtual environment 
generation realization apparatus according to the first 
embodiment. A CPU 1 executes programs writing the 
processing procedures of the first embodiment. A 
processing procedure storage device 2 stores the pro- 
grams writing the processing procedures. The programs 
may be pre-stored in the processing procedure storage 
device 2 but may be supplied from a storage medium 
such as an FD, CD-ROM. ROM, magnetic tape, or the 
like via an input/output interface (I/O IF) 8 (to be de- 
scribed later). A data storage device 3 holds data which 
are generated or input before, after, and during process- 
ing. A frame buffer 4 stores an image which is to be sup- 
plied to a display device to present an intermediate or 
final processing result. A window system 5 of an image 
display device displays the image stored in the frame 
buffer 4. A keyboard 6 is used for inputting characters 
and instructions during processing. The I/O IF 8 inter- 
faces with external devices to exchange data and the 
processing procedures. A bus 9 connects the above- 
mentioned devices. 

The data storage device 3 stores: 

shape model data d1, which include 

polygon data d11, 

surface attribute data d12, and 

free curved surface data d13, 

multi-viewpoint image data d2; 

light space data d3; and 

light space data arrangement data d4. 

Also, the processing procedure storage device 2 
stores: 

a full processing control routine pi ; 
a light space generation routine p2; 
a walk through control routine p3; and 
a drawing screen generation routine p4, which in- 
cludes 

a shape model drawing routine p41, 
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a drawing specification acquisition routine p42, 
an image generation routine p43 from a light 
space, and 

a fusion processing routine p44. 

Figs. 2A and 2B are flow charts showing the flow of 
the processing of the first embodiment. Programs that 
realize the processing of this flow chart are stored in the 
processing procedure storage device 2. 

In step S201 shape model data d1 and its surface 
attribute data d12, created in advance by a modeler ap- 
paratus which can create shape model data d1 of some 
computer graphics and its texture data, are input from 
an external device to the data storage device 3 via the 
I/O IF 8. The shape model data d1 includes polygon data 
d11 such as triangle patches, free curved surface data 
dl3 such as NURBS, and the like. The surface attribute 
data d12 includes the reflection characteristics and pat- 
tern of the corresponding shape data. In this case, tex- 
ture data representing a pattern is input as the surface 
attribute data d12. The texture data represents the sur- 
face pattern of polygon data d11, free curved suriace 
data d13, and the like, and describes the correspond- 
ence between the image data and shape. In step S202, 
multi-viewpoint image data d2 arc input from an external 
device to the data storage device 3 via the I/O IF 8. The 
multi-viewpoint image data d2 include a group of images 
taken from a plurality of positions. In step S203, light 
space data d3 is generated on the basis of the multi- 
viewpoint image data d2 in accordance with the light 
space generation routine p2 stored in the processing 
procedure storage device 2, and the generated data d3 
is stored in the data storage device 3. The multi-view- 
point image data d2 and light space data d3 input and 
generated in steps S202 and S203 are prepared in units 
of objects to be viewed, and a plurality of sets of data 
are prepared. The method of generating the light space 
data d3 in step S203 will be described below 

A technique for assuming a plane of z = 0 (to be 
referred to as a reference plane 31 hereinafter) on a co- 
ordinate system as shown in Fig. 3A, and expressing a 
three-dimensional space as a set of light rays that pass 
through this pfene has been proposed. As shown in Fig. 
4A, in this technique, an image that can be viewed from 
a viewpoint position P falling within a range Z > 0 in the 
three-dimensional space is equivalent to an image ob- 
tained by sampling only light rays that pass P from the 
set of light rays that pass through the reference plane 
31 Each light ray is expressed by a position (x, y) where 
it passes through the reference plane, angles $ and y 
the light ray makes with the x-and y-axes (Fig. 3B), time 
t at which the light ray passes through the plane, and a 
color (r, g, b) of the light ray. However, in practice, an 
object is assumed to be a still object and to have no 
parallax in the y-axis direction since the computation 
volume and data volume become huge if all these data 
are used Under this assumption, light rays are project- 
ed onto an x-u space using u - tan<t>. and when the light 



space is to be processed in the projected x-u space, light 
rays that pass a given point P define a linear path, as 
shown in Fig. 4B This path is given by: 

5 

x = X - Zu (1) 
u = tari(j) (2) 

10 

where (X. Z) represents the viewpoint position, and x 
represents the position of the intersection between each 
light ray and the x-axis on the x-u space. Also, $ repre- 
sents the angle which the light ray makes with the z-axis. 
'5 Subsequently, as shown in Fig. 5A, a linear path is 
calculated based on images obtained at a large number 
of viewpoint positions, and it is assumed that the x-u 
space is filled with these paths at high density. At this 
time, an image at a viewpoint position Q falling within 
20 the range z > 0 in Fig 5A can be obtained by calculating 
the paths of light rays that pass the position Q on the x- 
u space and acquiring the colors of light rays already . 
recorded on the paths, as shown in Fig. 5B. Generation 
of light space data in step S203 corresponds to mapping 
25 data of mutti-viewpoint image data on the x-u space in 
the above-mentioned processing. 

In step S204. arrangement data d4 indicating the 
arrangement of each light space data d3 with respect to 
the shape model data dl is input from an external device 
30 to the data storage device 3 via the I/O IF 8. The ar- 
rangement data d4 defines the position and direction of 
the reference plane which is set in the three-dimensional 
space upon generating light space data d3 and through 
which light passes. 
3S Fig. 6 is a schematic view showing the arrangement 
at that time. In Fig 5. a coordinate system transformed 
from the world coordinate system by a coordinate trans- 
formation T R is a light space coordinate system (a co- 
ordinate system in a Euclidean space where the refer- 
40 ence plane is present). On the other hand, a coordinate 
system transformed from the world coordinate system 
by a coordinate transformation T G is a shape model co- 
ordinate system, and a coordinate system further trans- 
formed trom the shape model coordinate system by a 
45 coordinate transformation T c is a camera coordinate 
system. 

Also, a view range that defines a scope within which 
an image can be generated based on light space data 
d3 is included in the arrangement data d4. 

so Fig. 7 is a schematic view associated with the gen- 
eration scope. The angle of the camera direction chang- 
es from mine to maxH, the x-coordinatc of the camera 
position changes from minx to maxx, and the z-coordi- 
nate thereof changes from minz tomaxz. 

5 5 In step S205. the size of the view screen and the 
angle of view of a virtual camera are initialized. 

In step S206 the initial position and direction of the 
viewer in a virtual space are designated. 
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Furthermore, in step S207, the drawing specifica- 
tion of the shape mode! data dl is calculated based on 
the angle of view of the virtual camera and the position 
and direction of the viewer in the virlual space upon 
viewing by the initialized screen size in accordance with 
the shape model drawing routine p41 in the processing 
procedure storage device 2, and a view image is gen- 
erated based on the shape model data. The generated 
image is written in the frame buffer 4 in step S208, and 
is displayed on a screen 5. Fig. 8 shows this display ex- 
ample. 

Moreover, in step S209, the distance between each 
light space data d3 and the viewpoint position is calcu- 
lated based on the arrangement data d4. In step S2095, 
the order of processing of the light space data d3 to be 
processed in steps S210 to S21 3 to be described below 
is determined on the basis ol the distances calculated 
in step S209. For example, in this case, the processing 
order from light space data d3 with larger distances is 
determined. In step S210, it is checked if processing is 
complete for all the light space data d3. If YES in step 
S210, the flow advances to step S214; otherwise, the 
flow advances to step S211. In step S211, it is checked 
if an image is to be generated based on the light space 
data d3 of interest of the arrangement data 64. For ex- 
ample, if the viewpoint position and direction fall within 
the view range included in the arrangement data d4, as 
shown in Fig. 9, it is determined in step S211 that an 
image is to be generated, and the flow advances to step 
S212; otherwise, it is determined in step S211 that an 
image is not to be generated, and the flow returns to 
step S210. In step S21 2, a drawing specification (which 
allows generation of the same view image as that drawn 
based on the shape model data in step S207) required 
for generating a view image from the light space data 
d3 of interest is calculated in accordance with the draw- 
ing specification acquisition routine p42 stored in the 
processing procedure storage device 2, and an image 
is generated from the light space data d3 of interest in 
accordance with the image generation routine p43 from 
light space data d3. The generated image is overwritten 
on the frame buffer 4 and is displayed on the screen 5, 
in step S2 13. 

If the processing in steps S21 0 to S213 is complete 
for all the light space data d3, the flow advances from 
step S210 to step S214. 

In step S214, it is checked if establishment of the 
view screen is to be changed. If YES in step S214, the 
flow advances to step S218, and the angle of view ratio 
and the window size are changed by broadening the 
window frame using a mouse. Thereafter, the flow re- 
turns to step S207, and for example, the display exam- 
pic shown in Fig. 8 is changed in step S208, as shown 
in Fig. 10. If establishment of the view screen is not to 
be changed, the flow advances to step S21 5. 

It is checked in step S215 if the viewpoint position 
and direction are to be changed If YES in step S215, 
the flow advances to step S219. and the viewpoint po- 



sition and direction are changed by performing forward 
and backward movements, clockwise and counterclock- 
wise turns, right and left movements, right and left look- 
ing, and the like using switches assigned to. e.g.. arrow 

5 keys while the window is being selected as the object to 
be controlled. Thereafter, the flow returns to step S207. 
and for example, the display example shown in Fig. 10 
is changed in step S208, as shown in Fig. 1 1 . If the view- 
point position and direction are not changed, the flow 

10 advances to step S2 1 6. 

In step S216, it is checked if the arrangement data 
d4 of light space data is to be changed. If YES in step 
S21 6, the flow advances to step S220, and the arrange- 
ment data d4 is changed by operating only the object 

is expressed by the light space using switches assigned 
to, e.g., arrow keys as in step S21 9. Thereafter, the flow 
returns to step S207, and for example, the display ex- 
ample shown in Fig. n is changed in step S208, as 
shown in Fig. 12. If the arrangement data d4 of light 

20 space data is not to be changed, the flow advances to 
step S217. 

These steps S214. S215, and S216 are attained by 
the walk through control routine p3 in the processing 
procedure storage device 2. In step S217, the control 

25 waits for an input indicating processing end. If an input 
indicating processing end is made, the processing ends; 
otherwise, the flow returns to step S21 4. 

Note that it is checked based on the position and 
direction of the arrangement data of the individual light 

30 space data d3 if the object expressed by the light space 
data d3 can be viewed from the position and direction 
of the viewer. If it is determined that the object can be 
viewed, the flow advances to step S209; otherwise, the 
flow advances to step S211 . 

35 With the above-mentioned steps, a three-dimen- 
sional space is generated using shape model data dl, 
a space is reconstructed as an image from the three- 
dimensional space in accordance with a drawing spec- 
ification at a given viewpoint, an image is generated 

to from light space data d3 in accordance with a drawing 
specification that allows the same view as the former 
drawing specification, and these images are fused, 
thereby generating a virtual environment for virtual re- 
ality and allowing the viewer to experience it. 

45 Fig. 9 or 12 shows a space actually generated by 
the first embodiment and an example of its view image. 
The geometric shape in the room and its surface pattern 
are expressed by the shape model, a rabbit doll is ex- 
pressed by the light space, and the image shown in Fig. 

50 9 or 1 2 is generated based on these images by the meth- 
od of this embodiment. 

In the first embodiment, two steps, i.e., step S202 
of inputting multi-viewpoint image data and step S203 
of generating light space data may be replaced by one 

5 5 step of reading light space data generated in advance 
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(b) Second Embodiment: 

In the first embodiment, as the actually taken im- 
age-based image generation processing, a view image 
is generated from light space data d3 in step S212. Al- 5 
ternatively, in the second embodiment, an image re- 
quired for display is generated from multi-viewpoint im- 
age data in place of the light space data d3 by the fol- 
lowing method. 

Fig. 1 3 shows the principle of reconstructing an im- w 
age from input multi-viewpoint image data. As shown in 
Fig. 1 3, as the conditions for phototaking multi-viewpoint 
images in the second embodiment, images must be tak- 
en by aligning cameras on a straight line perpendicular 
to the phototaking direction. Fig. 1 3 illustrates an object is 
1001, a line 1002 with projection viewpoints, which line 
connects the phototaking positions of input images, a 
virtual CCD surface 1003 of a virtual camera, and a vir- 
tual camera 1 004 set at the position (x\ z') of the viewer. 

The processing for generating an image from multi- 20 
viewpoint images will be explained below with reference 
to the flow chart in Fig. 14. In step S1101, a target line 
j is set at the head line of an image P, and a target pixel 
i is set at the pixel at the end of the line j. In step S1102, 
the position of an imago Q in the lino 1002 with projee- 25 
tion viewpoints corresponding to the i-th pixel of the line 
j of the pixel p is calculated. This position can be calcu- 
lated as follows. Assume that a certain point A is imaged 
at a pixel position Pi of the virtual camera at the view- 
point position P. Also, assume that Q represents the in- 30 
tersection between a line connecting the positions A and 
P, and the line 1002 with projection viewpoints. At this 
time, as can be seen from Figs. 3A and 3B, an object 
imaged at the pixel position Pi is equivalent to that im- 
aged at a pixel position Qi of the image taken at the view- 35 
point position Q. From the geometric limit condition in 
Figs. 3A and 3B, the x-coordmate of the viewpoint po- 
sition Q can be expressed by equation (3) below. In this 
case : the central pixel position of the scan line is as- 
sumed to be the 0th pixel. 4 o 



n of the image Q corresponding to the line j of the image 
P is calculated The method of calculating the line 
number n of the image Q will be described below with 
reference to Fig. 15. 

Fig. 15 shows the principle of correcting distortion 
of the image reconstructed by the above-mentioned 
method. Fig. 15 illustrates an object 1201, an image 
1 202 to be reconstructed at the viewpoint position P, and 
an image 1203 at the viewpoint position S on the line 
1002 with projection viewpoints 

A given point B in the object 1 201 will be examined. 
Assume that the point B is close to the y-axis, the z- 
coordinate values of the image 1 202 to be reconstructed 
at the viewpoint position P and the image 1203 at the 
viewpoint position S on the line 1002 with projection 
viewpoints are sufficiently large, or the z-coordmate val- 
ue Pz of the image 1202 to be reconstructed at the view- 
point position P is nearly equal to the z-coordinate value 
Sz of the image 1 203 at the viewpoint position S on the 
line 1002 with projection viewpoints. Al this lime, light 
rays coming from the point B are recorded on the m-th 
line in the image 1202 to be reconstructed at the view- 
point position P, and the n-th line in the image 1203 at 
the viewpoint position S on the line 1 002 with projection 
viewpoints. If d represents the pixel pitches of the virtual 
CCD 1003, f represents the focal length of the virtual 
camera 1004, and N represents the number of lines of 
the virtual CCD 1003, we have: 



Pz-tana = Sz-tan|5 



tana = d-(N/2 - m)/f 



tanfi = d-(N/2 - n)/f 
From equations (4), (5), and (6); 



(4) 



(5) 



(6) 



* x = x' +KJ-(g-z')/f (3) 

where d is the pixel pitch of the virtual camera 1004, f is 
the focal length, and g is the distance from the origin to 
the line 1002 with projection viewpoints. 

Similarly, an object imaged al a pixel position Pj of 
the virtual camera at the viewpoint position P is equiva- 
lent to that imaged at a pixel position Rj of an image at 
a viewpoint position R stored in the data storage device 
3. 

With this method, for example, when an image is 
reconstructed by copying the i-th pixel value of the line 
j of the image O to the i-th pixel of the line j of the image 
P. the reconstructed image is often distorted, i.e., the 
object in the image vertically stretches or shrinks In or- 
der to solve this problem, in step S1 103. the line number 



n = N/2 + (m - N/2)-Sz/Pz (7) 

Accordingly, the value of the m-th scan line of the 
image 1 202 to be reconstructed at the viewpoint position 
P is equivalent to that of the n-th scan line, given by 
equation (7), of the image 1 203 al the viewpoint position 
S on the line 1002 with projection viewpoints. Thus, in 
step S1104, the value of the i-th pixel of the line n of the 
image Q is copied to the i-th pixel ol the line j of the 
image P. With this processing, image distortion upon re- 
constructing the image at the viewpoint position, which 
is not on the line 1002 with projection viewpoints can be 
suppressed to some extent Thereafter, the flow ad- 
vances to step S1 105 to check if processing for all the 
pixels in the target line j is complete If YES in step 
S1105. the flow advances to step S1107; otherwise, the 
flow advances to step S1106. In step Si 106. the targel 



so 
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pixel i is moved to the right neighboring pixel, and the 
flow returns to step S1102 In step S1107. it is checked 
if the processing for all the lines in the image P is com- 
plete. If YES in step S1107. this subroutine ends: other- 
wise, the target line j is moved to the next line and the 
target pixel i is set at the left end of the line j in step 
S1108. Thereafter, the flow returns to step S1102. 

In this manner, if multi-viewpoint images taken at 
very small intervals on the line 1002 with projection 
viewpoints are obtained, the same processing based on 
the above-mentioned principle is repeated for alt the 
scan lines, thereby reconstructing an image at the view- 
point position which is not on the line 1002 with projec- 
tion viewpoints. 

(c) Third Embodiment: 

Fig. 16 is a block diagram showing the connections 
among the individual units of an image processing ap- 
paratus according to the third embodiment. 

The difference from Fig. 1 is that the data storage 
device 3 includes shape data 3a. surface attribute data 
3b (which is the same as the surface attribute data d12 
in Fig. 1 ), light space data 3c (which is the same as the 
light space data d3 in Fig. 1), a viewpoint position and 
direction 3d, and an angle of view and screen size 3e. 

Figs. 1 7 A and 1 7B are flow charts showing the flow 
of the processing of the third embodiment. The third em- 
bodiment will be described in detail below with reference 
to Figs. 1 7 A and 1 7B In order to generate a virtual en- 
vironment using geometric models, internal objects, ar- 
chitectures, and background that make up the virtual en- 
vironment need be created using, e.g. , a modeler appa- 
ratus which can create geometric shape data of some 
computer graphics and their surface attribute data. 

The existing technique can be used in this genera- 
tion processing. In step S401, it is selected whether a 
geometric shape model 3a and its surface attribute data 
3b are to be generated during this processing or data 
separately generated by another processing are to be 
read. If the geometric shape model 3a and its surface 
attribute data 3b are to be generated during this 
processing, thoy are generated in step S402; otherwise, 
data generated by another generation apparatus are 
read in step S403. Note that the shape data includes 
polygon data such as triangle patches, free curved sur- 
face data such as NURBS, and the like. The surface at- 
tribute data represents the material, reflection charac- 
teristics, pattern, and the like of the shape, and the pat- 
tern is input as texture data. The texture data is one rep- 
resenting the surface pattern of polygon data, free 
curved surface data, or the like, and describes image 
data of the pattern and the positional relationship be- 
tween the image and the shape to which the image is to 
be adhered. 

In step S404, it is selected whether light space data 
3c to be arranged in the virtual environment is to be gen- 
erated or light space data 3c generated in advance is to 



be read. If the data is to be generated, it is generated in 
slep S405 otherwise, it is read in step S406 

The method of generating the light space data 3c in 
step S405 has already been described in the para- 
s graphs of the first embodiment. 

The operation after the processing in step S405 or 
S406 is completed will be explained. 

In step S407, a three-dimensional world is built by 
arranging the generated data. Upon expressing the vir- 
10 tual environment, elements (coordinate transformation 
data, shape data, surface attribute data, illumination, 
and the like) that make up the virtual environment are 
expressed by a tree structure as in the conventional 
method (2). That is, a space, ground, architecture, room, 
is furniture, illumination, ornament, and the like that make 
up the three-dimensional space originally have a hier- 
archical nesting relationship thereamong. For example, 
an ornament on a table depends on the table arrange- 
ment so that it moves together with the table arrange- 

20 menl, and it is often convenient to arrange such orna- 
ment relative to the coordinate system of the table. For 
this reason, a data structure having hierarchical de- 
pendence on the arrangement is used. As a method of 
expressing such structure, a virtual environment is ex- 

2S pressed by an n ary tree structure. 

Fig. 20 shows an illustration example of a certain 
simple virtual environment. In the case of this figure, 
paying attention to a room, table, and sofa in the space, 
the room is described on a coordinate system C 2 trans- 

30 formed from a world coordinate system C 0 by a coordi- 
nate transformation T 2 . and the table and sofa in the 
room are respectively described on coordinate systems 
C 3 and C 4 transformed from the coordinate system C 2 
by coordinate transformations T 3 and T 4 . A pot on the 

35 table is described on a coordinate system C 5 trans- 
formed from the coordinate system C 3 by a coordinate 
transformation T 5 . Unlike in the conventional method 
(2), in the method of this embodiment, the light space 
data generated in step S403 or read in step S404 also 

40 expresses data as one element of this tree structure. In 
the case of Fig. 20, the light space data is arranged on 
the table This data is described on a coordinate system 
C 6 transformed from the coordinate system C 3 by coor- 
dinate transformation T 6 . In the existing technique, un- 
related data such as light space data which is different 
from the shape data cannot be described together. How- 
ever, in the method of this embodiment, the light space 
data is described as one element of the tree structure 
as in the shape data. With this method, an object having 

50 a complex shape is described using light space data, 
and simple data such as a room, a row of stores and 
houses, or the like is described using shape models, 
thus allowing the description oi the three-dimensional 
space by utilizing the features of the two different types 

55 of data. 

When the virtual environment shown in Fig. 20 is 
expressed by a typical tree structure, a tree shown in 
Fig 21 is obtained When the tree is expressed by text 
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data for the purpose of recording this tree structure data, 
the data formal shown in Fig 22 is obtained In Fig 22. 
"Separator" represents the head of a partial tree, and 
-Transform" represents a coordinate transformation 
Note that keywords "Separator" "Transform-, and the 
like are not limited to these specific ones as long as they 
can be distinguished from each other. Also, divisions us- 
ing T and ") " may be replaced by other symbols or char- 
acter strings. In this way, the light space data is de- 
scribed as one element that builds the tree structure. 

The operation after the processing in step S407 is 
completed will be described below. 

When data necessary for building the virtual envi- 
ronment remain ungenerated or unread, it is determined 
in step S408 that the control is to return to the pre- 
processing, and the flow returns to step S401. Other- 
wise, the flow advances to step S409 to check if the cur- 
rently generated virtual environment data are to be 
stored. If YES in step S409, the data are stored in step 
S4 1 0, otherwise, the flow advances to step S301 . When 
the data are stored in step S410, it is checked in step 
S411 if the processing is to end. If YES in step S411, 
the processing ends: otherwise, the flow also advances 
to step S301. 

A method of presenting the virtual environment ex- 
pressed by the tree structure by the above-mentioned 
method will be described in detail below. 

When a viewer views the virtual environment, the 
view image of 1he virtual environment from the viewpoint 
position must be drawn on the display screen. By tracing 
all the nodes of the n-ary tree shown in Fig. 21 by left 
node-depth preferential searching, the data of aff the 
nodes that build this tree are accessed. At this time, data 
is sequentially drawn on the display screen at the time 
of access to each node on the basis of information writ- 
ten in the node. The basic method has been described. 

In step S301. the view-screen is initialized. In step 
S302, the view point position and direction 3d are initial- 
ized. Subsequently, the flow advances to step S303 to 
start searching of the tree structure. In this case, the left 
node-depth preferential searching method is used (step 
S304). The searching for nodes is continued until this 
searching ends (step S305). 

If the contents of the current node indicate a coor- 
dinate transformation (step S306), the flow advances to 
step S307 to execute current coordinate transformation 
processing If no nodes corresponding to coordinate 
transformations are found so far, the coordinate trans- 
formation of the found node is set to be a current coor- 
dinate transformation as the basic coordinate transfor- 
mation of a partial tree deeper than the node; otherwise, 
the existing current coordinate transformation is multi- 
plied by the coordinate transformation of the newly 
found node to update the current coordinate transfor- 
mation for the partial tree deeper than the found node. 
Upon searching in the depth direction, the tree gives a 
current coordinate transformation to the next depth by 
passing the existing current coordinate transformation 



If there are no more deeper nodes, and the searching 
advances to a new shallower node, the current coordi- 
nate transformation which is to be updated by the next 
deeper node is not passed to the shallower node. In this 
5 case, the current coordinate transformation which was 
found by previous searching for the node of the tree at 
that depth is used 

If NO in step S306, the flow advances to step S306. 
If it is determined in step S308 that the contents of the 
'0 node indicate a surface attribute, the flow advances to 
step S309 to execute current surface attribute process- 
ing. When the node appears, the surface attribute of the 
node is set to be a current surface attribute as the basic 
surface attribute of a partial tree deeper than the node. 
'5 Upon searching the tree in the deeper direction, the cur- 
rent surface attribute is given to the next deeper node 
by passing the existing current surface attribute. If there 
are no more deeper nodes, and the searching advances 
to a new shallower node, the current surface attribute, 
20 which is lo be updated by the next deeper node, is not 
passed to the shallower node. In this case, the current 
surface attribute which was found by previous searching 
for the node of the tree at that depth is used. 

If NO in step S308. the flow advances to step S310. 
2S |f it is determined in stop S310 that the contents of the 
node indicate a geometric surface mode, the flow ad- 
vances to step S31 1 to execute geometric shape model 
drawing processing. In this processing, the coordinate 
transformation of the position, direction, and scaling of 
30 the shape model is performed using the current coordi- 
nate transformation, and a display image is drawn on 
the screen to be presented to the viewer as a two-di- 
mensional image by the conventional method (1 ) using 
the current surface attribute. At this time, a depth value 
35 storage map corresponding to the individual pixels of the 
presentation screen is prepared. Upon drawing a shape 
model in the drawing processing, the depth values of 
three-dimensional positions on the surface of the shape 
model at positions corresponding to the individual pixels 
4 o on the screen viewed from the viewpoint position are 
written in the corresponding pixels in the depth value 
storage map. In this case, if the depth values have al- 
ready been written in the depth value storage map by 
this processing, when a depth value obtained lor draw- 
ls mg a new image is larger than the already stored depth 
value, the value is not written in this pixel and the depth 
value storage map is not rewritten. 

If NO in step S31 0, the flow advances to step S31 2. 
If it is determined m step S312 that the contents of the 
^0 node indicate light space data, the flow advances to step 
S313 to execute drawing processing from light space 
data. In this processing, the coordinate transformation 
of the position, direction, scaling, and the like of light 
space data is performed using the current coordinate 
55 transformation. The position and direction, viewed from 
the reference plane of the light space data, of the view- 
point position are calculated A drawing specification 
used for generating an image to be generated from the 
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light space data under the view conditions is determined 
on the basis of the calculation results and the like, and 
an image to be displayed on the presentation screen is 
generated. At this time, as in the drawing processing 
from the geometric shape model, depth discrimination s 
in units of pixels on the display screen is performed. The 
distance between the viewpoint position and the posi- 
tion, on the reference plane, of each pixel of the light 
space data is used as the depth value of the pixel of the 
light space data, and this value is compared with a cor- io 
responding depth value stored in the depth value stor- 
age map. If the value of the pixel of interest is smaller 
than the corresponding value in the depth value storage 
map, the value of the pixel of the image generated from 
the light space data is overwritten on the value in the *s 
map, thereby updating the depth value storage map. 
Otherwise, the map is not updated. 

Upon completion of the processing in steps S307, 
S309, S3 1 1 , and S31 3, the flow returns to step S304 as 
the beginning of the tree searching processing. 20 

If it is determined in step S305 that tree searching 
has ended, viewing in the initial state is complete with 
the above-mentioned processing. Thereafter, it is 
checked in step S314 if the viewpoint position and di- 
rection 3d arc to be changed. If the viewer wants to walk 2s 
through the virtual environment, since he or she chang- 
es the viewpoint position and direction 3d in this step, 
the flow returns to step S303. If the viewpoint position 
and direction 3d are not changed, the flow advances to 
step S315 to check if the operation processing of the 30 
virtual environment is to be performed. This processing 
corresponds to, e.g.. an operation for moving an object 
in the virtual environment. If this processing is to be per- 
formed, the flow returns to step S303; otherwise, the 
flow advances to step S316. It is checked in step S316 3S 
if processing for changing the view screen such as the 
angle of view, screen size 3e, and the like is to be per- 
formed. If this processing is to be performed, the flow 
returns to step S303; otherwise, the flow advances to 
step S31 7. It ischecked in step S31 7 if all the processing -to 
operations are to end. If the processing is not to end, 
the flow returns to step S31 4. If some change instruction 
is input in stepS314, S315, or S3 16, the flow returns to 
step S303 to execute the drawing processing again. 

Fig. 23 shows a display example of the drawing •*$ 
screen of a given virtual environment. In Fig. 23, a rabbit 
doll is described by the light space data, and the room 
is described by the shape model. 

(d) Fourth Embodiment: so 

In the third embodiment, a display image is gener- 
ated from the light space data. In place of using the light 
space data, an image necessary for display can also be 
generated from multi-viewpoint images by the same ss 
method as in the second embodiment. 

The present invention can be applied to either a sys- 
tem built by a plurality of devices or an apparatus con- 



sisting of a single device. Needless to say. the present 
invention can also be applied to a case wherein the in- 
vention is attained by supplying a program to the system 
or apparatus. In this case, a storage medium that stores 
the program according to the present invention consti- 
tutes the present invention. By loading the program from 
the storage medium to the system or apparatus, the sys- 
tem or apparatus operates in accordance with the pre- 
determined method. 

As described in detail above, according to the 
above embodiments, the following effects are expected. 

An object with a very complex shape can be recon- 
structed by describing it on the basis of its way of view. 

Even when the images to be presented finally must 
be taken in advance, the data volume of images which 
must be prepared to artificially obtain an arbitrary view- 
point position and direction upon viewing can be re- 
duced greatly. 

Even when images from every positions in the 
three-dimensional space are to be presented, the data 
volume of required light space data can be reduced. 

Even when the viewpoint position is to be arbitrarily 
moved, phototaking of a panorama image can be facil- 
itated, and the data volume of panorama images to be 
prepared can be greatly reduced. 

Furthermore : by utilizing the characteristics of the 
conventional methods (1) to (5) which are originally in- 
dependent techniques, shape model data and an actu- 
ally taken image can be easily generated and fused to 
display an image with high reality 

Claims 

1 . An image processing method of generating and dis- 
playing a virtual environment, comprising: 

the model space rebuilding step of generating 
a three-dimensional space using shape model 
data, and rebuilding the space as an image 
from the three-dimensional space in accord- 
ance with a first drawing specification at a given 
viewpoint: 

the actually taken image-based image genera- 
tion step of generating an image from an arbi- 
trary viewpoint on the basis of an actually taken 
image in accordance with a second drawing 
specification that uses the same view condition 
as a view condition used by the first drawing 
specification; and 

the image fusion step of fusing the image rebuilt 
in the model space rebuilding step and the im- 
age generated in the actually taken imago- 
based image generation step. 

2. A method according to claim 1 , wherein the actually 
taken image-based image generation step uses 
light space data as the actually taken image 
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3. A method according to claim 1 , wherein the model 
space rebuilding step includes Ihe step of rebuilding 
the image using, as the first drawing specification, 
a perspective transformation derived from a posi- 
tion and direction of the viewpoint and an angle of 
view 

4. A method according to claim 1 . wherein the model 
space rebuilding step includes the step of rebuilding 
tnc image using, as the first drawing specification, 
an orthogonal projection transformation derived 
from a position and direction of the viewpoint. 

5. A method according to claim 1, wherein the image 
Usion step includes the step of fusing two drawn 
imnrjcs by displaying an image drawn based on the 
sfMoc model data on a screen first, and thereafter, 
overwriting an image drawn based on the actually 
i iken image on the screen. 

6. A Method according to claim 1, wherein the shape 
r>odel data in the model space rebuilding step is not 
d.iin that expresses only a geometric shape but is 
d.ii.i iilso including a surface characteristic and pat- 
cm of the geometric shape. 

7. A method according to claim 1 , wherein the actually 
i ikon image-based image generation step compris- 

tno image reconstruction step of reconstructing 
.in image from a plurality of images taken from a 
(.iut iiiiy of positions. 

8 a m.Mhod according to claim 7, wherein the actually 
i tKi n image-based image generation step compris- 
• Mm? distortion correction step of correcting distor- 
: < of an image upon reconstructing the image in 
i'« wringe reconstruction step. 

9 Ar t ir rvige processing apparatus for generating and 
ci* - .( Living a virtual environment, comprising: 



20 

1 0. An apparatus according to claim 9. wherein said ac- 
tually taken image-based image generation means 
uses light space data as the actually taken image. 

5 11. An apparatus according to claim 9, wherein said 
model space rebuilding means includes rebuilds 
the image using, as the first drawing specification, 
a perspective transformation derived from a posi- 
tion and direction of the viewpoint and an angle of 

io view. 

12. An apparatus according to claim 9, wherein said 
model space rebuilding means rebuilds the image 
using, as the first drawing specification, an orthog- 

'5 onal projection transformation derived from a posi- 
tion and direction of the viewpoint. 

13. An apparatus according to claim 9. wherein said im- 
age fusion means fuses two drawn images by dis- 

20 playing an image drawn based on the shape model 
data on a screen first, and thereafter, overwriting an 
image drawn based on the actually taken image on 
the screen. 

25 14. An apparatus according to claim 9, wherein the 
shape model data used by said model space re- 
building means is not data that expresses only a ge- 
ometric shape but is data also including a surface 
characteristic and pattern of the geometric shape. 

30 

1 5. An apparatus according to claim 9. wherein said ac- 
tually taken image-based image generation means 
comprises image reconstruction means for recon- 
structing an image from a plurality of images taken 
from a plurality of positions. 

16. An apparatus according to claim 15, wherein said 
actually taken image-based image generation 
means comprises distortion correction means for 
correcting distortion of an image upon reconstruct- 
ing the image by said image reconstruction means. 

17. An image processing method of generating a virtual 
environment, comprising: 



model space rebuilding means for generating 
n mrtee-dimensional space using shape model 
dnta. and rebuilding the space as an image 
trom the three-dimensional space in accord- -*s 
nnce with a first drawing specification at a given 
viewpoint; 

nctually taken image-based image generation 
moans for generating an image from an arbi- 
tinry viewpoint on the basis of an actually taken 
image in accordance with a second drawing 
specification that uses the same view condition 
;ic a view condition used by the first drawing 
spocilication; and 

image fusion means for fusing the image rebuilt 
by said model space rebuilding means and the 
image generated by said actually taken image- 
based image generation means 
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the first description step of describing a shape 
of a three-dimensional object and a shape of a 
three-dimensional space on the basis of infor- 
mation including a three-dimensional position, 
a three-dimensional direction, and a surface 
color; 

the second description step of describing a 
three-dimensional object and a three-dimen- 
sional space on the basis of an actually taken 
image taken from an arbitrary position; and 
the generation step of generating the virtual en- 
vironment by (using the shapes described in the 
first description step and Ihe three-dimensional 
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object and the three-dimensional space de- 
scribed in the second description step 

18. A method according to claim 17. wherein the gen- 
eration step includes the step of generating the vir- 
tual environment by fusing the shapes described in 
the first description step and the three-dimensional 
object and the three-dimensional space described 
in the second description step by a tree structure 
which includes the shapes and the objects as con- 
stituent elements, 

1 9. A method according to claim 1 8, further comprising: 

the access step of searching and accessing all 
the constituent elements of the tree structure 
by a three search method; 
the drawing step of performing drawing suitable 
for each of the constituent elements accessed 
during the access step; and 
the changing step of changing a view condition 
of the virtual environment generated in the gen- 
eration step. 

20. A method according to any one of claims 17 to 19, 
wherein the first and second description steps per- 
form the descriptions using light space data. 

21 . A method according to claim 1 9, wherein when the 
constituent element to be accessed is a shape mod- 
el, the drawing step comprises: 

the step of calculating a positional relationship 
between a viewpoint position and direction and 
the shape model; and 

the step of displaying an image on a display 
screen using a perspective transformation or 
an orthogonal projection transformation on the 
basis of the calculated relationship. 

22. A method according to claim 1 9, wherein when the 
constituent element to be accessed is light space 
data, the drawing step comprises: 

the step of calculating a positional relationship 
between a viewpoint position and direction and 
the light space data; and 
the step of generalingan image lobe presented 
from the light space data on the basis of the 
calculated relationship. 

23. A method according to claim 22, wherein when the 
constituent clement to be accessed is light space 
data, the drawing step further comprises: 

the step of calculating a distance between an 
arrangement position of the light space data 
and the viewpoint position; and 



the step of discriminating depths of the constit- 
uent elements on the basis of the calculation 
result used as a depth value of the light space 
data 

5 

24. A method according to claim 22, wherein the draw- 
ing step comprises: 

the step of preparing a depth value storage map 
10 having a size for one screen, 

when the constituent element to be accessed 
is a shape model, the drawing step further com- 
prises: 

the step of generating a drawing image to be 
is drawn on a display screen; 

the step ol calculating distances between a 
viewpoint and pixels of the shape model corre- 
sponding to pixel positions on the display 
screen; and 

20 the step of displaying the drawing image by 

storing the distances in the depth value storage 
map in units of pixels when no depth values are 
stored in the depth value storage map, and dis- 
playing the drawing image on the display 

25 screen by storing the distances in the depth val- 

ue storage map in units of pixels when the 
depth values are stored in the depth value stor- 
age map and when the calculated distances are 
compared with the corresponding depth values 

30 stored in the depth value storage map in units 

of pixels and the depth value is smaller than the 
corresponding distance, and 
when the constituent element to be accessed 
is light space data, the drawing step further 

35 comprises: 

the step of generating an image to be presented 
on the basis of the light space data: 
the depth value approximation step of calculat- 
ing a distance between a viewpoint position and 

40 an arrangement position of the light space data 

and determining the calculation result as an ap- 
proximate depth value of the light space data; 
and 

the step of displaying the generated image on 
45 the display screen by stonng the approximate 

depth value in the depth value storage map in 
units of pixels when no depth value is stored in 
the depth value storage map, and displaying a 
pixel of the generated pixel on the display 
50 screen by storing the approximate depth value 

in the depth value storage map in units of pixels 
when the depth value is stored in the depth val- 
ue storage map and when the approximate 
depth value is compared with the depth value 
55 in the depth value storage value in units of pix- 

els and the approximate depth value is smaller 
than the stored depth value 
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25. An image processing apparatus for generating a vir- 
tual environment, comprising 

first description means for describing a shape 
of a three-dimensional object and a shape of a s 
three-dimensional space on the basis of infor- 
mation including a three-dimensional position, 
a three-dimensional direction, and a surface 
color; 

second description means for describing a w 
three-dimensional object and a three-dimen- 
sional space on the basis of an actually taken 
image taken from an arbitrary position; and 
generation means for generating the virtual en- 
vironment by fusing the shapes described by is 
said first description means and the three-di- 
mensional object and the three-dimensional 
space described by said second description 
means. 

20 

26. An apparatus according to claim 25, wherein said 
generation means generates the virtual environ- 
ment by fusing the shapes described by said first 
description means and the three-dimensional ob- 
ject and the three-dimensional space described by 2S 
said second description means by a tree structure 
which includes the shapes and the objects as con- 
stituent elements. 

27. An apparatus according to claim 26. further com- 30 
prising 

access means for searching and accessing all 
the constituent elements of the tree structure 
by a three search method; 35 
drawing means for performing drawing suitable 
for each of the constituent elements accessed 
during accessing by said access means; and 
changing means for changing a view condition 
of the virtual environment generated by said 40 
generation means. 

28. An apparatus according to any one of claims 25 to 
27, wherein said first and second description means 
perform the descriptions using light space data. -ts 

29. An apparatus according to claim 27, wherein when 
the constituent element to be accessed is a shape 
model, said drawing means comprises. 

so 

means for calculating a positional relationship 
between a viewpoint position and direction and 
the shape model; and 

means for displaying an image on a display 
screen using a perspective transformation or ss 
an orthogonal projection transformation on the 
basis of Ihe calculated relationship 



30. An apparatus according to claim 27, wherein when 
the constituent element to be accessed is light 
space data, said drawing means comprises: 

means for calculating a positional relationship 
between a viewpoint position and direction and 
the light space data; and 
means for generating an image to be presented 
from the light space data on the basis of the 
calculated relationship. 

31. An apparatus according to claim 29, wherein when 
the constituent element to be accessed is light 
space data, said drawing means further comprises 

means for calculating a distance between an 
arrangement position of the light space data 
and the viewpoint position, and 
means for discriminating depths of the constit- 
uent elements on the basis of the calculation 
result used as a depth value of the light space 
data. 

32. An apparatus according to claim 29, wherein said 
drawing means comprises: 

means for preparing a depth value storage map 

having a size for one screen, 

when the constituent element to be accessed 

is a shape model, said drawing means further 

comprises: 

means for generating a drawing image to be 
drawn on a display screen; 
means for calculating distances between a 
viewpoint and pixels of the shape model corre- 
sponding to pixel positions on the display 
screen; and 

means for displaying the drawing image by 
storing the distances in the depth value storage 
map in units of pixels when no depth values are 
stored in the depth value storage map, and dis- 
playing the drawing image on the display 
screen by storing the distances in the depth val- 
ue storage map in units ol pixels when the 
depth values are stored in the depth value stor- 
age map and when the calculated distances are 
compared with the corresponding depth values 
stored in the depth value storage map in units 
of pixels and the depth value is smaller than the 
corresponding distance, and 
when the constituent element to be accessed 
is light space data, said drawing means further 
comprises: 

means for generating an image to be presented 
on the basts of the light space data 
depth value approximation means for calculat- 
ing a distance between a viewpoint position and 
an arrangement position ol the light space data 
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and determining the calculation result as an ap- 
proximate depth value of the light space data; 
and 

means for displaying the generated image on 
the display screen by storing the approximate 5 
depth value in the depth value storage map in 
units of pixels when no depth value is stored in 
the depth value storage map, and displaying a 
pixel of the generated pixel on the display 
screen by storing the approximate depth value 10 
in the depth value storage map in units of pixels 
when the depth value is stored in the depth val- 
ue storage map and when the approximate 
depth value is compared with the depth value 
in the depth value storage value in units of pix- is 
els and the approximate depth value is smaller 
than the stored depth value. 
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